{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Advanced usage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook shows some more advanced features of `skorch`. More examples will be added with time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Table of contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* [Setup](#Setup)\n",
    "* [Callbacks](#Callbacks)\n",
    "  * [Writing your own callback](#Writing-a-custom-callback)\n",
    "  * [Accessing callback parameters](#Accessing-callback-parameters)\n",
    "* [Working with different data types](#Working-with-different-data-types)\n",
    "  * [Working with dicts](#Working-with-dicts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "import torch.nn.functional as F"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "torch.manual_seed(0);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A toy binary classification task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We load a toy classification task from `sklearn`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.datasets import make_classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X, y = make_classification(1000, 20, n_informative=10, random_state=0)\n",
    "X = X.astype(np.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1000, 20), (1000,), 0.5)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape, y.shape, y.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Definition of the `pytorch` classification `module`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define a vanilla neural network with two hidden layers. The output layer should have 2 output units since there are two classes. In addition, it should have a softmax nonlinearity, because later, when calling `predict_proba`, the output from the `forward` call will be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from skorch.net import NeuralNetClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class ClassifierModule(nn.Module):\n",
    "    def __init__(\n",
    "            self,\n",
    "            num_units=10,\n",
    "            nonlin=F.relu,\n",
    "            dropout=0.5,\n",
    "    ):\n",
    "        super(ClassifierModule, self).__init__()\n",
    "        self.num_units = num_units\n",
    "        self.nonlin = nonlin\n",
    "        self.dropout = dropout\n",
    "\n",
    "        self.dense0 = nn.Linear(20, num_units)\n",
    "        self.nonlin = nonlin\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        self.dense1 = nn.Linear(num_units, 10)\n",
    "        self.output = nn.Linear(10, 2)\n",
    "\n",
    "    def forward(self, X, **kwargs):\n",
    "        X = self.nonlin(self.dense0(X))\n",
    "        X = self.dropout(X)\n",
    "        X = F.relu(self.dense1(X))\n",
    "        X = F.softmax(self.output(X), dim=-1)\n",
    "        return X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Callbacks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Callbacks are a powerful and flexible way to customize the behavior of your neural network. They are all called at specific points during the model training, e.g. when training starts, or after each batch. Have a look at the `skorch.callbacks` module to see the callbacks that are already implemented."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Writing a custom callback"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although `skorch` comes with a handful of useful callbacks, you may find that you would like to write your own callbacks. Doing so is straightforward, just remember these rules:\n",
    "* They should inherit from `skorch.callbacks.Callback`.\n",
    "* They should implement at least one of the `on_`-methods provided by the parent class (e.g. `on_batch_begin` or `on_epoch_end`).\n",
    "* As argument, the `on_`-methods first get the `NeuralNet` instance, and, where appropriate, the local data (e.g. the data from the current batch). The method should also have `**kwargs` in the signature for potentially unused arguments.\n",
    "* *Optional*: If you have attributes that should be reset when the model is re-initialized, those attributes should be set in the `initialize` method."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is an example of a callback that remembers at which epoch the validation accuracy reached a certain value. Then, when training is finished, it calls a mock Twitter API and tweets that epoch. We proceed as follows:\n",
    "* We set the desired minimum accuracy during `__init__`.\n",
    "* We set the critical epoch during `initialize`.\n",
    "* After each epoch, if the critical accuracy has not yet been reached, we check if it was reached.\n",
    "* When training finishes, we send a tweet informing us whether our training was successful or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from skorch.callbacks import Callback\n",
    "\n",
    "\n",
    "def tweet(msg):\n",
    "    print(\"~\" * 60)\n",
    "    print(\"*tweet*\", msg, \"#skorch #pytorch\")\n",
    "    print(\"~\" * 60)\n",
    "\n",
    "\n",
    "class AccuracyTweet(Callback):\n",
    "    def __init__(self, min_accuracy):\n",
    "        self.min_accuracy = min_accuracy\n",
    "\n",
    "    def initialize(self):\n",
    "        self.critical_epoch_ = -1\n",
    "\n",
    "    def on_epoch_end(self, net, **kwargs):\n",
    "        if self.critical_epoch_ > -1:\n",
    "            return\n",
    "        # look at the validation accuracy of the last epoch\n",
    "        if net.history[-1, 'valid_acc'] >= self.min_accuracy:\n",
    "            self.critical_epoch_ = len(net.history)\n",
    "\n",
    "    def on_train_end(self, net, **kwargs):\n",
    "        if self.critical_epoch_ < 0:\n",
    "            msg = \"Accuracy never reached {} :(\".format(self.min_accuracy)\n",
    "        else:\n",
    "            msg = \"Accuracy reached {} at epoch {}!!!\".format(\n",
    "                self.min_accuracy, self.critical_epoch_)\n",
    "\n",
    "        tweet(msg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we initialize a `NeuralNetClassifier` and pass your new callback in a list to the `callbacks` argument. After that, we train the model and see what happens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "net = NeuralNetClassifier(\n",
    "    ClassifierModule,\n",
    "    max_epochs=10,\n",
    "    lr=0.02,\n",
    "    warm_start=True,\n",
    "    callbacks=[AccuracyTweet(min_accuracy=0.7)],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  epoch    train_loss    valid_acc    valid_loss     dur\n",
      "-------  ------------  -----------  ------------  ------\n",
      "      1        \u001b[36m0.6908\u001b[0m       \u001b[32m0.5950\u001b[0m        \u001b[35m0.6842\u001b[0m  0.2184\n",
      "      2        \u001b[36m0.6876\u001b[0m       0.5950        \u001b[35m0.6815\u001b[0m  0.0480\n",
      "      3        \u001b[36m0.6853\u001b[0m       \u001b[32m0.6100\u001b[0m        \u001b[35m0.6789\u001b[0m  0.0447\n",
      "      4        0.6882       0.5950        \u001b[35m0.6769\u001b[0m  0.0459\n",
      "      5        \u001b[36m0.6780\u001b[0m       0.6000        \u001b[35m0.6743\u001b[0m  0.0468\n",
      "      6        \u001b[36m0.6730\u001b[0m       0.6100        \u001b[35m0.6717\u001b[0m  0.0467\n",
      "      7        \u001b[36m0.6664\u001b[0m       \u001b[32m0.6150\u001b[0m        \u001b[35m0.6698\u001b[0m  0.0450\n",
      "      8        0.6670       0.6100        \u001b[35m0.6670\u001b[0m  0.0445\n",
      "      9        0.6667       \u001b[32m0.6300\u001b[0m        \u001b[35m0.6646\u001b[0m  0.0446\n",
      "     10        \u001b[36m0.6624\u001b[0m       \u001b[32m0.6350\u001b[0m        \u001b[35m0.6624\u001b[0m  0.0501\n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n",
      "*tweet* Accuracy never reached 0.7 :( #skorch #pytorch\n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<class 'skorch.net.NeuralNetClassifier'>[initialized](\n",
       "  module_=ClassifierModule(\n",
       "    (dense0): Linear(in_features=20, out_features=10)\n",
       "    (dropout): Dropout(p=0.5)\n",
       "    (dense1): Linear(in_features=10, out_features=10)\n",
       "    (output): Linear(in_features=10, out_features=2)\n",
       "  ),\n",
       ")"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net.fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oh no, our model never reached a validation accuracy of 0.7. Let's train some more (this is possible because we set `warm_start=True`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     11        0.6647       \u001b[32m0.6500\u001b[0m        \u001b[35m0.6598\u001b[0m  0.0447\n",
      "     12        \u001b[36m0.6573\u001b[0m       \u001b[32m0.6650\u001b[0m        \u001b[35m0.6575\u001b[0m  0.0486\n",
      "     13        \u001b[36m0.6458\u001b[0m       \u001b[32m0.6700\u001b[0m        \u001b[35m0.6549\u001b[0m  0.0485\n",
      "     14        0.6528       \u001b[32m0.6750\u001b[0m        \u001b[35m0.6525\u001b[0m  0.0449\n",
      "     15        0.6476       0.6700        \u001b[35m0.6502\u001b[0m  0.0322\n",
      "     16        0.6483       0.6750        \u001b[35m0.6476\u001b[0m  0.0325\n",
      "     17        0.6514       \u001b[32m0.6800\u001b[0m        \u001b[35m0.6452\u001b[0m  0.0339\n",
      "     18        \u001b[36m0.6365\u001b[0m       \u001b[32m0.6850\u001b[0m        \u001b[35m0.6422\u001b[0m  0.0329\n",
      "     19        \u001b[36m0.6335\u001b[0m       \u001b[32m0.7000\u001b[0m        \u001b[35m0.6390\u001b[0m  0.0290\n",
      "     20        0.6381       \u001b[32m0.7100\u001b[0m        \u001b[35m0.6363\u001b[0m  0.0309\n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n",
      "*tweet* Accuracy reached 0.7 at epoch 19!!! #skorch #pytorch\n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<class 'skorch.net.NeuralNetClassifier'>[initialized](\n",
       "  module_=ClassifierModule(\n",
       "    (dense0): Linear(in_features=20, out_features=10)\n",
       "    (dropout): Dropout(p=0.5)\n",
       "    (dense1): Linear(in_features=10, out_features=10)\n",
       "    (output): Linear(in_features=10, out_features=2)\n",
       "  ),\n",
       ")"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net.fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, the validation score exceeded 0.7. Hooray!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accessing callback parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Say you would like to use a learning rate schedule with your neural net, but you don't know what parameters are best for that schedule. Wouldn't it be nice if you could find those parameters with a grid search? With `skorch`, this is possible. Below, we show how to access the parameters of your callbacks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To simplify the access to your callback parameters, it is best if you give your callback a name. This is achieved by passing the `callbacks` parameter a list of *name*, *callback* tuples, such as:\n",
    "\n",
    "    callbacks=[\n",
    "        ('scheduler', LearningRateScheduler)),\n",
    "        ...\n",
    "    ],\n",
    "    \n",
    "This way, you can access your callbacks using the double underscore semantics (as, for instance, in an `sklearn` `Pipeline`):\n",
    "\n",
    "    callbacks__scheduler__epoch=50,\n",
    "\n",
    "So if you would like to perform a grid search on, say, the number of units in the hidden layer and the learning rate schedule, it could look something like this:\n",
    "\n",
    "    param_grid = {\n",
    "        'module__num_units': [50, 100, 150],\n",
    "        'callbacks__scheduler__epoch': [10, 50, 100],\n",
    "    }\n",
    "    \n",
    "*Note*: If you would like to refresh your knowledge on grid search, look [here](http://scikit-learn.org/stable/modules/grid_search.html#grid-search), [here](http://scikit-learn.org/stable/auto_examples/model_selection/grid_search_text_feature_extraction.html), or in the *Basic_Usage* notebok."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below, we show how accessing the callback parameters works our `AccuracyTweet` callback:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "net = NeuralNetClassifier(\n",
    "    ClassifierModule,\n",
    "    max_epochs=10,\n",
    "    lr=0.1,\n",
    "    warm_start=True,\n",
    "    callbacks=[\n",
    "        ('tweet', AccuracyTweet(min_accuracy=0.7)),\n",
    "    ],\n",
    "    callbacks__tweet__min_accuracy=0.6,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  epoch    train_loss    valid_acc    valid_loss     dur\n",
      "-------  ------------  -----------  ------------  ------\n",
      "      1        \u001b[36m0.7139\u001b[0m       \u001b[32m0.5500\u001b[0m        \u001b[35m0.6933\u001b[0m  0.0488\n",
      "      2        \u001b[36m0.6916\u001b[0m       \u001b[32m0.5950\u001b[0m        \u001b[35m0.6873\u001b[0m  0.0473\n",
      "      3        \u001b[36m0.6829\u001b[0m       0.5800        \u001b[35m0.6814\u001b[0m  0.0613\n",
      "      4        \u001b[36m0.6671\u001b[0m       \u001b[32m0.6050\u001b[0m        \u001b[35m0.6718\u001b[0m  0.0542\n",
      "      5        \u001b[36m0.6670\u001b[0m       \u001b[32m0.6200\u001b[0m        \u001b[35m0.6639\u001b[0m  0.0554\n",
      "      6        \u001b[36m0.6622\u001b[0m       \u001b[32m0.6350\u001b[0m        \u001b[35m0.6546\u001b[0m  0.0567\n",
      "      7        \u001b[36m0.6371\u001b[0m       \u001b[32m0.6550\u001b[0m        \u001b[35m0.6429\u001b[0m  0.0450\n",
      "      8        \u001b[36m0.6293\u001b[0m       \u001b[32m0.6700\u001b[0m        \u001b[35m0.6312\u001b[0m  0.0445\n",
      "      9        \u001b[36m0.6170\u001b[0m       0.6650        \u001b[35m0.6200\u001b[0m  0.0425\n",
      "     10        0.6204       \u001b[32m0.6750\u001b[0m        \u001b[35m0.6119\u001b[0m  0.0423\n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n",
      "*tweet* Accuracy reached 0.6 at epoch 4!!! #skorch #pytorch\n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<class 'skorch.net.NeuralNetClassifier'>[initialized](\n",
       "  module_=ClassifierModule(\n",
       "    (dense0): Linear(in_features=20, out_features=10)\n",
       "    (dropout): Dropout(p=0.5)\n",
       "    (dense1): Linear(in_features=10, out_features=10)\n",
       "    (output): Linear(in_features=10, out_features=2)\n",
       "  ),\n",
       ")"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net.fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, by passing `callbacks__tweet__min_accuracy=0.6`, we changed that parameter. The same can be achieved by calling the `set_params` method with the corresponding arguments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<class 'skorch.net.NeuralNetClassifier'>[initialized](\n",
       "  module_=ClassifierModule(\n",
       "    (dense0): Linear(in_features=20, out_features=10)\n",
       "    (dropout): Dropout(p=0.5)\n",
       "    (dense1): Linear(in_features=10, out_features=10)\n",
       "    (output): Linear(in_features=10, out_features=2)\n",
       "  ),\n",
       ")"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net.set_params(callbacks__tweet__min_accuracy=0.75)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  epoch    train_loss    valid_acc    valid_loss     dur\n",
      "-------  ------------  -----------  ------------  ------\n",
      "     11        \u001b[36m0.5845\u001b[0m       \u001b[32m0.7000\u001b[0m        \u001b[35m0.6016\u001b[0m  0.0459\n",
      "     12        \u001b[36m0.5831\u001b[0m       \u001b[32m0.7050\u001b[0m        \u001b[35m0.5915\u001b[0m  0.0463\n",
      "     13        0.5854       \u001b[32m0.7200\u001b[0m        \u001b[35m0.5788\u001b[0m  0.0440\n",
      "     14        \u001b[36m0.5582\u001b[0m       0.7150        \u001b[35m0.5729\u001b[0m  0.0428\n",
      "     15        0.5601       0.7150        \u001b[35m0.5692\u001b[0m  0.0547\n",
      "     16        \u001b[36m0.5468\u001b[0m       \u001b[32m0.7250\u001b[0m        \u001b[35m0.5662\u001b[0m  0.0461\n",
      "     17        \u001b[36m0.5333\u001b[0m       \u001b[32m0.7300\u001b[0m        \u001b[35m0.5583\u001b[0m  0.0458\n",
      "     18        0.5592       0.7200        \u001b[35m0.5555\u001b[0m  0.1286\n",
      "     19        \u001b[36m0.5295\u001b[0m       0.7300        \u001b[35m0.5488\u001b[0m  0.0546\n",
      "     20        \u001b[36m0.5232\u001b[0m       0.7300        \u001b[35m0.5428\u001b[0m  0.0433\n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n",
      "*tweet* Accuracy never reached 0.75 :( #skorch #pytorch\n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<class 'skorch.net.NeuralNetClassifier'>[initialized](\n",
       "  module_=ClassifierModule(\n",
       "    (dense0): Linear(in_features=20, out_features=10)\n",
       "    (dropout): Dropout(p=0.5)\n",
       "    (dense1): Linear(in_features=10, out_features=10)\n",
       "    (output): Linear(in_features=10, out_features=2)\n",
       "  ),\n",
       ")"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net.fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with different data types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Working with dicts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The standard case"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "skorch has built-in support for dictionaries as data containers. Here we show a somewhat contrived example of how to use dicts, but it should get the point across. First we create data and put it into a dictionary `X_dict` with two keys `X0` and `X1`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X, y = make_classification(1000, 20, n_informative=10, random_state=0)\n",
    "X0, X1 = X[:, :10].astype(np.float32), X[:, 10:].astype(np.float32)\n",
    "X_dict = {'X0': X0, 'X1': X1}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When skorch passes the dict to the pytorch module, it will pass the data as keyword arguments to the forward call. That means that we should accept the two keys `XO` and `X1` in the forward method, as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class ClassifierWithDict(nn.Module):\n",
    "    def __init__(\n",
    "            self,\n",
    "            num_units0=50,\n",
    "            num_units1=50,\n",
    "            nonlin=F.relu,\n",
    "            dropout=0.5,\n",
    "    ):\n",
    "        super(ClassifierWithDict, self).__init__()\n",
    "        self.num_units0 = num_units0\n",
    "        self.num_units1 = num_units1\n",
    "        self.nonlin = nonlin\n",
    "        self.dropout = dropout\n",
    "\n",
    "        self.dense0 = nn.Linear(10, num_units0)\n",
    "        self.dense1 = nn.Linear(10, num_units1)\n",
    "        self.nonlin = nonlin\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        self.output = nn.Linear(num_units0 + num_units1, 2)\n",
    "\n",
    "    # NOTE: We accept X0 and X1, the keys from the dict, as arguments\n",
    "    def forward(self, X0, X1, **kwargs):\n",
    "        X0 = self.nonlin(self.dense0(X0))\n",
    "        X0 = self.dropout(X0)\n",
    "\n",
    "        X1 = self.nonlin(self.dense1(X1))\n",
    "        X1 = self.dropout(X1)\n",
    "\n",
    "        X = torch.cat((X0, X1), dim=1)\n",
    "        X = F.relu(X)\n",
    "        X = F.softmax(self.output(X), dim=-1)\n",
    "        return X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As long as we keep this in mind, we are good to go."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "net = NeuralNetClassifier(ClassifierWithDict, verbose=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<class 'skorch.net.NeuralNetClassifier'>[initialized](\n",
       "  module_=ClassifierWithDict(\n",
       "    (dense0): Linear(in_features=10, out_features=50)\n",
       "    (dense1): Linear(in_features=10, out_features=50)\n",
       "    (dropout): Dropout(p=0.5)\n",
       "    (output): Linear(in_features=100, out_features=2)\n",
       "  ),\n",
       ")"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net.fit(X_dict, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Working with sklearn `FunctionTransformer` and `GridSearch`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import FunctionTransformer\n",
    "from sklearn.model_selection import GridSearchCV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "sklearn makes the assumption that incoming data should be numpy/sparse arrays or something similar. This clashes with the use of dictionaries. Unfortunately, it is sometimes impossible to work around that for now (for instance using skorch with `BaggingClassifier`). Other times, there are possibilities.\n",
    "\n",
    "When we have a preprocessing pipeline that involves `FunctionTransformer`, we have to pass the parameter `validate=False` so that sklearn allows the dictionary to pass through:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pipe = Pipeline([\n",
    "    ('do-nothing', FunctionTransformer(validate=False)),\n",
    "    ('net', net),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "     steps=[('do-nothing', FunctionTransformer(accept_sparse=False, func=None, inv_kw_args=None,\n",
       "          inverse_func=None, kw_args=None, pass_y='deprecated',\n",
       "          validate=False)), ('net', <class 'skorch.net.NeuralNetClassifier'>[initialized](\n",
       "  module_=ClassifierWithDict(\n",
       "    (dense0): Linear(in_features=10, out_features=50)\n",
       "    (dense1): Linear(in_features=10, out_features=50)\n",
       "    (dropout): Dropout(p=0.5)\n",
       "    (output): Linear(in_features=100, out_features=2)\n",
       "  ),\n",
       "))])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe.fit(X_dict, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When trying a grid or randomized search, it is not that easy to pass a dict. If we try, we will get an error:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "param_grid = {\n",
    "    'net__module__num_units0': [10, 25, 50], \n",
    "    'net__module__num_units1': [10, 25, 50],\n",
    "    'net__lr': [0.01, 0.1],\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "grid_search = GridSearchCV(pipe, param_grid, scoring='accuracy', verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found input variables with inconsistent numbers of samples: [2, 1000]\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    grid_search.fit(X_dict, y)\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The error above occurs because sklearn gets the length of the input data, which is 2 for the dict, and believes that is inconsistent with the length of the target (1000). \n",
    "\n",
    "To get around that, skorch provides a helper class called `SliceDict`. It allows us to wrap our dictionaries so that they also behave like a numpy array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from skorch.helper import SliceDict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_slice_dict = SliceDict(X0=X0, X1=X1)  # X_slice_dict = SliceDict(**X_dict) would also work"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The SliceDict shows the correct length, shape, and is sliceable across values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Length of dict: 2, length of SliceDict: 1000\n",
      "Shape of SliceDict: (1000,)\n"
     ]
    }
   ],
   "source": [
    "print(\"Length of dict: {}, length of SliceDict: {}\".format(len(X_dict), len(X_slice_dict)))\n",
    "print(\"Shape of SliceDict: {}\".format(X_slice_dict.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Slicing the SliceDict slices across values: SliceDict(**{'X0': array([[-0.96583462, -2.18907046,  0.16985609,  0.81384557, -3.37520909,\n",
      "        -2.14305973, -0.39585084,  2.94195771, -2.19106054,  1.24439669],\n",
      "       [-0.45476699,  4.33976793, -0.48572844, -4.8843298 , -2.8836503 ,\n",
      "         2.60972047, -1.95287597, -0.09192174,  0.07970932, -0.08938338]], dtype=float32), 'X1': array([[ 0.04351204, -0.51509613, -0.86073655, -1.10971689,  0.31839254,\n",
      "        -0.82319731, -1.05630398, -0.89645284,  0.37592441, -1.08496511],\n",
      "       [-0.60726726, -1.06743085,  0.48804346, -0.50230557,  0.55743027,\n",
      "         1.01592004, -1.99535823,  2.90304255, -0.97392982,  2.17533231]], dtype=float32)})\n"
     ]
    }
   ],
   "source": [
    "print(\"Slicing the SliceDict slices across values: {}\".format(X_slice_dict[:2]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this, we can call `GridSearch` just as expected:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 18 candidates, totalling 54 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  54 out of  54 | elapsed:   35.0s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=None, error_score='raise',\n",
       "       estimator=Pipeline(memory=None,\n",
       "     steps=[('do-nothing', FunctionTransformer(accept_sparse=False, func=None, inv_kw_args=None,\n",
       "          inverse_func=None, kw_args=None, pass_y='deprecated',\n",
       "          validate=False)), ('net', <class 'skorch.net.NeuralNetClassifier'>[initialized](\n",
       "  module_=Classi...es=50)\n",
       "    (dropout): Dropout(p=0.5)\n",
       "    (output): Linear(in_features=100, out_features=2)\n",
       "  ),\n",
       "))]),\n",
       "       fit_params=None, iid=True, n_jobs=1,\n",
       "       param_grid={'net__module__num_units0': [10, 25, 50], 'net__module__num_units1': [10, 25, 50], 'net__lr': [0.01, 0.1]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score=True,\n",
       "       scoring='accuracy', verbose=1)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.fit(X_slice_dict, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.75,\n",
       " {'net__lr': 0.1,\n",
       "  'net__module__num_units0': 25,\n",
       "  'net__module__num_units1': 50})"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.best_score_, grid_search.best_params_"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
